NFL Big Data Bowl 2022

Introduction

Jim Kelly once said, “There’s not one person who’s even been through our training camp who could cover him. Nobody could cover him one-on-one”. Jim Kelly wasn’t referring to Andre Reed (HoF WR) or James Lofton, he was referring to Steve Tasker who only had 51 career receptions over 14 years in the NFL. At 5’9’’, Steve Tasker was arguably the best gunner in NFL history making 7 Pro Bowls and was the only special teams player to ever earn Pro Bowl MVP (1993). A gunner is a position on the special teams who’s sole role is to break free from defenders during a punt to tackle the returner and limit their return yardage. The goal of my analysis is to build on the initial research done by Michael Lopez build on the initial research done by Michael Lopez to add specific metrics to evaluate Gunners in the NFL.

We will be using 2020 tracking and scouting information to create four metrics to evaluate Gunner effectiveness during punts. The metrics are:
1. Tackle Opportunity Probability Added
2. Expected Gunner Distance at Punt Reception Under Expected
3. Expected Return Yards Under Expected
4. Expected Return Yards Under Exepcted (Returned Punts)

These four metrics will help to measure which gunners put themselves in the best position to limit yardage from the returner, which is their ultimate job.

Tackle Opportunity Probability Added

The animated play below is 3 yard punt return where two Gunners, Nsimba Webster (#14) and David Long (#25), are vying to either limit return yardage, get a tackle opportunity or force a fair catch. In this example, Webster here is credited with the tackle. This visual helps to portray a common punt so we will use this play to illustrate gunner effectiveness. First, we want to define tackle opportunity (or involvement), which we’ve defined as either being the primary tackler, assist tackler, or even having a missed tackle. Regardless, involvement in tackling, whether converted or not, we believe is important to measure.

If we break down the tackle opportunity by frame, Webster had ~30% probability of a tackle at snap, ~28% at punt reception. Despite the ~0.02 probability added, we can see a wide variation in probability between that timeframe. At the lowest, the probability dropped 18% and the high we see a 30% probability (at the snap) and had ~25.3% average probability through the duration of the play.

The reason for Webster’s drop in probability to 18% was that he had beat his vises (#29) at the snap, #18 on the receiving team recognizes this and sealed the block. The increase back in probability is due to the fact that by gaining the attention of the second defender, he actually created a log jam at the receiver thus limiting his potential return yardage and increasing his probability of a tackle opportunity (Again, Webster was credited with the tackle).

This scenario helps to paint a good picture of how critical the Gunner’s are in tackling the returner.

In the scenario above, we showed results of the Tackle Opportunity Probability model, which can be used to evaluate Gunner’s ability to put themselves in a better position for a tackle opportunity from snap to punt reception.

For creating our model, we used NFL tracking data and filtered for the following:
1. Punt Plays
2. Using PFF data, filtering for those who were assigned as Gunner’s
3. Filtering for catchable punts (filters out touchbacks)

Filtering out touchbacks helps us to model for plays where a gunner could have had a high probability of a tackle even in cases of a fair catch (where a high probability tackle opp is).

In building our model, we included the following variables for consideration:
-Gunner’s field position and direction variance
-Gunner’s speed variance
-Distance from Line of Scrimmage (LOS)
-Distance from Ball
-Total distance travelled
-Avg. separation from key closest players
-Position Types on the field (# of Gunners, # of Vises)

Our final classification model is a Gradient Boosted Machine (GBM) using down sampling, 10 cross fold validations, and 5 repeats to properly train the model and improve the performance of the model. Gradient Boosted Machines is a machine learning technique that uses multiple learning algorithms with the goal of “boosting” or reducing bias and variance. It essentially convert weak learning models in a “boosted” or stronger one. The custom hyperparemter sampling techniques added helps to iterate through our decision trees to create the most accurate model possible.

In our evaluating our model, we got a mean ROC score of 80.2%, mean sensitivity score of 66.1%, and a mean specificity score of 80.6%. Essentially, our model is better at calculating non tackle opportunities than calculating tackle opportunities.

Another way to interpret our model is through feature importance (Gini impurity), which helps to describe the decision tree nodes in order of relative importance. Basically, this helps to show what factors contribute the most to our prediction model. We can see that a player’s distance to the ball, their position variance per second, and speed variance are the biggest factors that contribute to player’s tackle opportunity probability. It’s interesting to see that the number of total Vises on the receiving team doesn’t have much importance in a gunner’s Tackle Probability.

What this boils down to is the commitment to the angle of pursuit on the punt. We can see this by visualizing all of Webster’s punt routes for received punts (Green representing tackles & Blue representing missed tackles). In the first few seconds of the snap, you see a rather clean angle being formed and that’s due to the gunner’s commitment to the angle.

Pulling it all together, we see that Webster had the most Tackle Opps of all Gunners in 2020 for returnable punts but a lower TPA (Tackle Probability Added from Snap to Punt Reception). A big part of his ability to create more Tackle Opportunities is relatively high avg speed, higher variance in speed (i.e. adjusting route to returner), and higher separation from other vises. Contrary to Webster in strategy and also of note is Justin Bethel, who had the one of the highest avg. TPA due to his high avg speed, high avg max speed, and low variance (usually straight shot in route). Bethel was the only gunner to eclipse 20 mph in his average max speed per punt. Lastly, we see Matthew Slater listed high here and that’s important to note considering he’s been to 9 Pro Bowls as a gunner.

RK1	Name	Team		TPA2	Tackle Opp.3	Punts	Avg speed (MPH)	Avg Max Speed (MPH)4	Avg Speed Variance (MPH)	Avg Vise Separation (Yds)
1.0	James Pierre	PIT		0.118	8	46	13.09	19.24	0.37	2.29
2.0	Justin Bethel	NE		0.116	5	41	15.14	20.19	0.35	2.31
3.0	Matt Slater	NE		0.115	3	40	13.69	19.66	0.35	2.26
4.0	J.T. Gray	NO		0.112	4	50	14.33	19.59	0.40	2.53
5.0	Danny Johnson	WAS		0.104	3	54	13.02	18.62	0.36	1.89
6.0	Brandon Wilson	CIN		0.103	8	49	14.15	19.66	0.35	2.04
7.0	Mack Hollins	MIA		0.101	3	48	13.82	19.33	0.40	2.18
8.0	Charles Washington	ARI		0.094	7	41	14.04	19.53	0.39	2.32
9.5	Trent Sherfield	ARI		0.093	6	47	14.02	19.38	0.38	4.05
9.5	Justin Layne	PIT		0.093	5	59	13.39	19.28	0.38	2.22
11.0	Cordarrelle Patterson	CHI		0.091	5	49	13.05	18.48	0.37	2.22
12.0	Nsimba Webster	LA		0.087	9	50	14.18	19.78	0.39	2.41
1 RK = Avg TPA rank 2 TPA = Avg. Tackle Probability Added (Avg Delta of Probability at Punt Reception and Probability at Snap 3 Tackle Opp = Counts involvement in a tackle: Tackle, Missed or Assisted 4 Avg of Highest Speed Reached per punt play

Expected Gunner Distance and Expected Return Yards

TPA is a helpful metric to evaluate what Gunner’s put themselves in the best position for a tackle on a punt play. However, as we see know not every punt play is returned by the returned. To help to evaluate players in all punt players, we are creating two more models to evaluate which Gunner’s get to the returner at punt reception and which Gunners are the best at limiting ball return yardage.

TThe distribution chart below illustrates that there is vast difference amongst a gunner’s ability to get to the ball for a given punt play type, which indicates that some gunners are better at getting to the returner and thus limiting return yardage.

This leads us to the creation of the Expected Distance to Ball & Expected Return Yards models, which were both trained using a Random Forest and optimized using a 10 fold repeated cross validation (5x). This helps to resample the dataset to prevent any training bias. The purpose of the first model is to predict, starting at punt, what a gunner’s distance to ball will be. The prediction at punt reception will be used to compare to where the gunner’s actual distance is at punt reception. The purpose of the second model, is to understand how a player’s proximity, speed, and other variables (same used to predict tackle probability) can influence a returner’s return yardage. The prediction at punt reception can used as the official metric to evaluate how a gunner can limit return yardage.

To evaluate the accuracy and effectiveness of the models, R-squared, Root Mean Square Error (RMSE), and Mean Absolute Error (MAE) were chosen as our main metrics. R-squared is the how close the predictions are to the actual values from a proportional or percentage value, which for these two models we got over 98% goodness of fit. RMSE is a measurement of error rate over all observations and MAE is mean error. We can see below that both these models have less than a 2% error rate (RMSE) and an average observed difference of less than 1 yard (MAE).

Ball Distance Metrics:
-R-squared: 0.9901339
-RMSE: 0.9497339
-MAE: 0.5204672

Return Yards Metrics:
-R-squared: 0.9833967
-RMSE: 1.6052536
-MAE: 0.6868127

There are two applications of these models. The first is shown below as a way to augment film study by showing per punt how the gunner was able to get to the retuner. Our sample play from prior has had the predictions added through the addition of 3 lines. The first line (in red) shows the expected return yardage from the punt returner. The second two lines (in grey) show the predicted distance each gunner will be the punt returner by the point of reception. We can see how the predictions change as the play develops.

Gunner’s Return Yard Under Expected

The second application of the model is to evaluate what their prediction was at punt reception versus what actually occurred. The average delta will be created as a metric for evaluation of Gunners. This leads to the creation of Return Yards Under Expected (RYUE) and Ball Distance Under Expected (BDUE). The Gunners who were able to limit the most amount return yardage (RYUE) and get closer to the ball than expected (BDUE) would ultimately be best at their position. Similarly to TPA, we see that Slater (perennial pro-bowler) and Bethel are near the top of the ranks here. What’s interesting about Webster, is he ranks high in his ability to get to the ball (BDUE) but doesn’t put himself in the best position to limit return yardage, which was a similar theme for his lower TPA compared to other Gunners. J.T. Gray below has consistent in BDUE (~7 yards closer to the ball than expected on average), and his ability to limit return yardage.

RK	Name	Team		RYUE1	BDUE2	RYUE (Non Fair Catch)3	RYUE (Fair Catch)4	Punts
1.0	J.T. Gray	NO		-0.446	-7.069	-0.487	-0.405	50
2.0	Matt Slater	NE		-0.410	-6.464	-0.370	-0.492	40
3.0	Justin Bethel	NE		-0.370	-5.797	-0.338	-0.432	41
4.0	Charles Washington	ARI		-0.256	-6.255	-0.156	-0.428	41
5.0	Danny Johnson	WAS		-0.233	-7.917	-0.056	-0.535	54
6.0	Trent Sherfield	ARI		-0.163	-6.247	-0.026	-0.366	47
7.0	Brandon Wilson	CIN		-0.160	-6.953	-0.084	-0.343	49
8.5	Justin Layne	PIT		-0.154	-7.885	0.075	-0.488	59
8.5	James Pierre	PIT		-0.154	-7.856	0.041	-0.458	46
10.0	Cordarrelle Patterson	CHI		0.117	-6.533	0.404	-0.338	49
11.0	Nsimba Webster	LA		0.148	-7.004	0.538	-0.488	50
12.0	Mack Hollins	MIA		0.315	-6.499	0.795	-0.419	48
1 RYUE = Avg. Return Yards Under Expected 2 BDUE = Avg. Distance to Ball Under Expected 3 Filters RYUE for punts that were not fair caught 4 Filters RYUE for punts that were fair caught

Application of Models & Future Opportunities

These models were specifically designed with the intent to evaluate Gunner’s effectiveness in punt plays but could easily be extended to evaluate other players on the punt play (Vises, others in pursuit). By understanding how Gunner’s are able to change speed, direction, and angle, other punting players can use the learnings to fine tune their role within the punt play. Another application of the models is to help to augment film review. The visualizations prior showed tackle probability, distance to ball, and expected return yards all by frame. These visualizations could easily be replicated for other punt plays as a way for players to understand their attack to a returner.

Source Code

Personal Twitter